Skip to content

Add kueue component #6358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

gbenhaim
Copy link
Member

Install kueue and the plugins for supporting tekton pipelineruns and external admission.
Install only as part of the development overlay and only on OCP cluster with version >= 4.16 (the kueue operator doesn't work on earlier version).

ATM, the index image for the RH productized version of Kueue isn't yet ready, because of that it's being installed using the operator-framework cli.

@gbenhaim gbenhaim requested review from hugares, sadlerap and filariow May 11, 2025 08:23
@openshift-ci openshift-ci bot requested review from elsony and johnmcollier May 11, 2025 08:23
@gbenhaim gbenhaim force-pushed the add-kueue-component branch from b487ab0 to 15622d6 Compare May 11, 2025 08:59
@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

@gbenhaim
Copy link
Member Author

/test

Copy link

openshift-ci bot commented May 12, 2025

@gbenhaim: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test appstudio-e2e-tests
/test appstudio-load-test

The following commands are available to trigger optional jobs:

/test appstudio-hac-e2e-tests
/test appstudio-upgrade-tests
/test konflux-e2e-v416-optional

Use /test all to run the following jobs that were automatically triggered:

pull-ci-redhat-appstudio-infra-deployments-main-appstudio-e2e-tests
pull-ci-redhat-appstudio-infra-deployments-main-appstudio-upgrade-tests

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@gbenhaim
Copy link
Member Author

/test appstudio-e2e-tests
/test appstudio-upgrade-tests
/test konflux-e2e-v416-optional

Copy link
Contributor

@sadlerap sadlerap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few questions and comments, but mostly looks fine by me.

roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to grant this service account admin rights on the namespace?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used to run a job which installs the operator in the namespace, usually only admin has the rights to install operators. It would be difficult to pinpoint the exact permissions it needs on the namespace (and even than those would be broad).

Comment on lines +112 to +118
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "200m"
memory: "256Mi"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we set the requests and the limits to be the same to avoid pod evictions?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it was done on the initContainer, but not on this container?

@gbenhaim gbenhaim force-pushed the add-kueue-component branch from 448b7d4 to dad9ff1 Compare May 13, 2025 06:36
@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

1 similar comment
@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

@gbenhaim gbenhaim force-pushed the add-kueue-component branch from 994ff0f to dc47888 Compare May 29, 2025 08:27
@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

1 similar comment
@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

@gbenhaim gbenhaim force-pushed the add-kueue-component branch from dc47888 to c8b44fe Compare May 29, 2025 11:23
@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

@gbenhaim gbenhaim force-pushed the add-kueue-component branch from c8b44fe to 881f983 Compare May 29, 2025 12:10
@gbenhaim
Copy link
Member Author

/test konflux-e2e-v416-optional

gbenhaim added 15 commits June 4, 2025 20:30
- Add the manifests for deploying Kueue using the RH operator.
  Since wasn't released yet (and it doesn't have an index image),
  use operator-sdk for installing it.

- Add the ArgoCD AppSet for Kueue.

Signed-off-by: Gal Ben Haim <[email protected]>
The configmap is used for telling the admission controller if it
need to admit workloads. This is a temporary solution until the
integration with alert manager will by implemented.

Signed-off-by: Gal Ben Haim <[email protected]>
It's not possible to install it on earlier versions.

Signed-off-by: Gal Ben Haim <[email protected]>
Signed-off-by: Gal Ben Haim <[email protected]>
Configure the Pipelineruns queues on the cluster.

- LocalQueue for each tenant - created by Kyverno
- A global cluster queue configured with admission check
  provided by kueue-external-admission.

Signed-off-by: Gal Ben Haim <[email protected]>
- tekton-kueue
- kueue-external-admission

Signed-off-by: Gal Ben Haim <[email protected]>
Adding the defaults so Argo won't think that resource is out of sync.

Signed-off-by: Gal Ben Haim <[email protected]>
Signed-off-by: Gal Ben Haim <[email protected]>
Signed-off-by: Gal Ben Haim <[email protected]>
@gbenhaim gbenhaim force-pushed the add-kueue-component branch from 4848002 to e264b2b Compare June 4, 2025 17:30
@gbenhaim
Copy link
Member Author

gbenhaim commented Jun 4, 2025

/test konflux-e2e-v416-optional

1 similar comment
@gbenhaim
Copy link
Member Author

gbenhaim commented Jun 4, 2025

/test konflux-e2e-v416-optional

gbenhaim added 2 commits June 4, 2025 22:07
When a Workload doesn't have cpu or memory requests, if there is a limit range
in the namespace, Kueue will assign the requests from it to the Workload.
We don't care about the cpu/memory for PipelineRuns, thus setting a high value for both.

Signed-off-by: Gal Ben Haim <[email protected]>
@gbenhaim gbenhaim force-pushed the add-kueue-component branch from 6efeaf5 to f15ead2 Compare June 4, 2025 19:25
@gbenhaim
Copy link
Member Author

gbenhaim commented Jun 4, 2025

/test konflux-e2e-v416-optional

3 similar comments
@gbenhaim
Copy link
Member Author

gbenhaim commented Jun 5, 2025

/test konflux-e2e-v416-optional

@gbenhaim
Copy link
Member Author

gbenhaim commented Jun 5, 2025

/test konflux-e2e-v416-optional

@gbenhaim
Copy link
Member Author

gbenhaim commented Jun 5, 2025

/test konflux-e2e-v416-optional

Signed-off-by: Gal Ben Haim <[email protected]>
@gbenhaim
Copy link
Member Author

gbenhaim commented Jun 5, 2025

/test konflux-e2e-v416-optional

Copy link

openshift-ci bot commented Jun 5, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gbenhaim, hugares

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hugares
Copy link
Contributor

hugares commented Jun 5, 2025

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants